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ABSTRACT: In this paper we show that the optical flow, a 2-D field that 
can be associated with the variation of the image brightness pattern, and the 
2-D motion field, the projection on the image plane of the 3-D velocity field 
of a moving scene, are in general different, unless very special conditions are 
satisfied. The optical flow, therefore, is ill-suited for computing structure 
from motion and for reconstructing the 3-D velocity field, problems that 
require an accurate estimate of the 2-D motion field. We then suggest a 
different use of the optical flow. We argue that stable qualitative properties 
of the 2-D motion field give useful information about the 3-D velocity field 
and the 3-D structure of the scene, and that they can be usually obtained 
from the optical flow. To support this approach we show how the (smoothed) 
optical flow and 2-D motion field, interpreted as vector fields tangent to flows 
of planar dynamical systems, may have the same qualitative properties from 
the point of view of the theory of structural stability of dynamical systems. 
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1. Introduction 

A key task for many vision systems is to extract information from a sequence 
of images. This information can be useful to solve important problems such 
as recovering the 3-D velocity field, or segmenting the image into parts cor- 
responding to different moving objects, or reconstructing the 3-D structure 
of surfaces. The recovery of the 2-D motion field, (that we define as the 
projection on the image plane of the 3-D velocity field) is thought to be an 
essential step in the solution of these problems. The data available, how- 
ever, are temporal variations in the brightness pattern. These variations are 
usually associated with a perceived motion field, called optical flow (Gibson, 
1950; Fennema and Thompson, 1979; Horn and Schunck, 1981). In order 
to recover the 2-D motion field, the assumption that the 2-D motion field 
and the optical flow coincide has often been made. It must be noted though, 
that this assumption is clearly satisfied only in the case in which variations 
in the brightness pattern correspond to features of the visible, 3-D surfaces. 
In fact several authors have developed algorithms to reconstruct the 2-D mo- 
tion field from optical flow data defined only at locations of features in the 
image (Hildreth, 1984a,b; Waxman, 1986). Examples in which this assump- 
tion does not hold are known (Horn, 1986), but they have been regarded as 
pathological cases. As a matter of fact, algorithms that deal with the recov- 
ery of the 2-D motion field from dense optical flow data have been proposed, 
with the more or less explicit assumption that the two fields are the same 
(Horn and Schunck, 1981; Nagel, 1984; Kanatani, 1985). 

In this paper we show that the optical flow and the motion field are in 
general different, unless very special conditions are satisfied. We explicitly 
compute the difference between their normal components (the component 
along the direction of the gradient) under broad assumptions. We show 



that they are arbitrarily close where the image gradient is sufficiently strong. 
Hence, feature-based matching algorithms that rely on edges of various types 
(including texture edges) are more appropriate than point-to-point ones to 
solve problems that rely on accurate recovery of the 2-D motion field, such 
as structure from motion. One may then ask, what is the optical flow for? In 
the second part of the paper we suggest that meaningful information about 
the 3-D velocity field and the 3-D structure can be obtained from qualitative 
properties of the 2-D motion field. We then argue that this information can 
be retrieved directly from the optical flow or its normal components. We 
describe a specific approach that exploits results from the theory of stability 
of dynamical systems. A more detailed analysis of this approach will be 
presented in a forthcoming paper by V. Torre and coworkers. 

The paper is divided in two parts. In the first, we define the problem 
and we state explicitly the assumptions that we have used. In particular, we 
consider in detail how image irradiance can be related to scene radiance in 
the case of a scene consisting of non-lambertian surfaces. We describe, then, 
a method that allows us to show that the optical flow and the motion field are 
almost always different. We analytically compute the difference between the 
normal components of the two fields assuming, first, the lambertian model 
of reflectance and then a more realistic one for arbitrary rigid motion of 
a generic surface. We also calculate how this difference depends on the 
image gradient and the 3-D velocity of moving objects. In the second part 
we show how both the optical flow and the motion field can be processed 
to become vector fields tangent to flows of dynamical systems. The optical 
flow then, can be considered as a perturbed motion field under the conditions 
determined in the first part. Results from the theory of stability of dynamical 
systems suggest that qualitative, stable properties of the motion field hold 
for the optical flow. We sketch some example of these properties and how 



they can be used in a description of the 3-D velocity field. We finally discuss 
briefly some connections with biological systems. 



2. Preliminaries 



In this chapter we review the definitions of motion field and optical flow, and 
we state the assumptions that we used throughout the paper. In particular, 
we consider in detail how image irradiance can be related to scene radiance 
in the case of a scene consisting of non-lambertian surfaces. 

2.1. Definitions 



Let us define notations and summarize definitions that will be useful in what 
follows. For more details on the geometry of perspective projection see Ap- 
pendix Al. Throughout the following we will assume, if it is not otherwise 
stated, that any expression can be differentiated as many times as nedeed. 

Let 

/ 
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p f + x-n 



(x-(x-n)n) (2.1.1) 



be the equation defining the projection of a generic point on the image plane, 
where x p = (x p ,y p ,0) is the position vector of the projected point, x — 
(x,y,z) is the position vector of the point, n is the unit vector normal to 
the image plane (projection plane) and / is the focal length (see Figure 1). 
Notice that the origin O is on the image plane, the focus of projection F is 
located at (0,0, — /), and /n + x is the vector pointing from F to the point. 




Figure 1. The geometry of perspective projection. 



The motion field v p can be obtained differentiating (2.1.1) with respect 
to the time. If v = dx/dt we have 1 
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Notice that in (2.1.2) v p is given in terms of x and v, position and velocity 

of the moving points in the scene, which are not known. 

Let E = E(x p ,y p ,t) be the image irradiance, that is the intensity of 
light at the point (x p ,y p ) of the image plane at the time t. If V p is the 



gradient with respect to the image coordinates, then 

dE dE 
dt 
Now if 
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*It can be easily shown that the perspective projection of the 3-D velocity vector 
is equal to the velocity of the projected point on the image plane, since both the 
vector are defined in terms of infinitesimal. This is not true for a generic, finite 
vector 
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Therefore, if (2.1.4) holds, the projection of the motion field along the di- 
rection of the gradient can be given in terms of derivatives of the image 
irradiance (which can be computed). In what follows, this component will 
be called vj_, or the normal component; thus 

v ± = ^ V 'f I>* (2.1.6) 

||V p jC/|| ||V p /i|| 

Equation (2.1.5) can be interpretated as an instance of the well-known 
aperture problem (Marr and Ullman, 1981; Horn and Schunck, 1981): that 
is the information available at each point of a sequence of frames is only 
the component of the motion field along the direction of the image gradient. 
To recover a full and unique motion field, some other constraint is needed: 
Horn and Schunck (1981), for example, showed that there is only one 2- 
D field whose normal component coincides with (2.1.6) and which is the 
smoothest of all possible ones. Examples for which (2.1.5) is not true are 
well known (Horn and Schunck, 1981). Consider, for instance, a rotating 
sphere with no texture on it {i.e. with uniform albedo) under arbitrary, 
fixed illumination. Since the image irradiance at each image location does 
not change with time, the left-hand side of (2.1.5) is identically equal to zero, 
while the right-hand side is different from zero almost everywhere. Notice 
that keeping the sphere fixed and moving the light source (2.1.5) is again 
wrong. In this case, however, the left-hand-side is different from zero while 
u_L is zero everywhere. In both cases the perceived motion in the image is 
different from the motion field. It is worthwhile, then, to introduce a new 
field, called the minimal optical flow, related to the perceived motion in the 
image, and not necessarily equal to the normal component of the motion field. 
Notice that the perceived motion in the examples above agrees qualitatively 



with the left-hand-side of (2.1.5). Indeed, the optical flow in the first case 
is identically equal to zero, while in the second is different from zero almost 
everywhere. Therefore let us define the normal component Op of the optical 
flow as: 

OF = _^L^iL (a . L7) 

T7 F V7 F v 

|| V p £,|| || \ pCj\\ 

Hence, with respect to this definition, the minimal optical flow and the nor- 
mal component of the motion field are always directed along the gradient 
and they coincide if and only if (2.1.4) holds. 

Remark: in the literature, it is usually assumed that (2.1.4) holds. As a 
consequence, the normal components of the motion field and of the optical 
flow are the same and the latter can be used as a constraint to recover the 
2-D motion field. 

2.2. Scene Radiance and Image Irradiance 

Let us review briefly some definitions of photometry and make explicit the 
constraints under which the image irradiance is related to the scene radiance. 
The image irradiance E is the power per unit area of light at each point 
(x p ,y p ) of the image plane: thus E — E(x p ,y p ). The scene radiance L 
is the power per unit area of light that can be thought emitted by each 
point of a surface S in the scene in a particular direction. This surface 
can be fictitious, or it may be the actual radiating surface of a light source, 
or the illuminated surface of a solid. The scene radiance can be thought 
as a function of the point of the surface and of the direction in space. If 
(a, b) are intrinsic coordinates of the surface and (a, (3) polar coordinates 
determining a direction in space with respect to the normal to the surface, 
we can write L = L(a,b,a,(3). Given the scene radiance it is possible, in 
principle, to compute the expected image irradiance. For example in the 




Figure 2. Scene radiance and image irradiance in the pinhole approximation: the 
image irradiance at the point (x p ,y p ) is given by the scene radiance at the point 
(a, b) on the surface in the direction of the line connecting the two points and 
passing through the pinhole Pfj . 



case of pinhole camera approximation, that is assuming that the camera has 
an infinitesimally small aperture, the image irradiance at a point (x p ,y p ) is 
proportional to the scene radiance at the point (a, 6) on the surface in the 
direction of the pinhole, say (a ,/? ), where (x p ,y p ), (a,b) and the pinhole 
lie on the same line (see Figure 2). Therefore we have 



E(x p (a,b),y p (a,b)) = L(a,b,a u ,f3 u ) 



(2.2.1) 



where (x p (a,b),y p (a,b)) is the image point that lies on the line connecting 
(a, 6) to the pinhole. In practice, however, the aperture of any real optical 
device is finite and not very small (ultimately to avoid diffraction effects): 
thus (2.2.1) does not generally hold. Assuming that the surface is lambertian, 
i.e. L(a,b,a,(3) = L(a,b), that there are not losses within the system and 
that the angular aperture (on the image side) is small it can be proved (Born 
and Wolf, 1959) that 

E(x p (a,b),y p (a,b)) = L(a, b)U cos 4 <p (2.2.2) 

where is the solid angle corresponding to the angular aperture and <p is 
the angle between the principal ray (that is the ray passing through the 
center of the aperture) and the optical axis. With the further assumption 
that the aperture is much smaller than the distance of the viewed surface, 
the lambertian hypothesis can be relaxed to give (Horn and Sjoberg, 1979) 

E{x p (a,b),y p {a,b)) = L(a, b, a ,/3 )fl cos 4 <p (2.2.3) 

where a u and 0° are the polar coordinates of the direction of the principal 
ray. It must be pointed out that (2.2.3) holds if L is continuous with respect 
to a and /?. In what follows we will assume that this is the case. Furthermore, 
we will assume that the optical system has been calibrated so that (2.2.3) 
can be rewritten as (2.2.1). Finally, notice that 

n r _ r .da db. 

V p E. Vp = VsL- (-,-), (2.2.4) 

where V 5 is the gradient with respect to the surface coordinates, since dif- 
ferentiating (2.2.1) we have 

V p E-{dx p ,dy p ) = V s L(da,db). (2.2.5) 
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3. Minimal optical flow and Motion Field 

We describe a general method that allows us to show that the minimal optical 
flow and the normal component of the motion field are almost always differ- 
ent, or equivalently that (2.1.4) does not hold. We compute the difference 
between the normal components of the two fields, assuming first the Lamber- 
tian model of reflectance and then a more realistic one for pure translation, 
pure rotation and general rigid motion of a generic surface. It turns out 
that the two fields are equal only under very special conditions, which can 
be explicitly stated. We also show that the difference is smaller where the 
image gradient is stronger, justifying the use of feature-based algorithms. Of 
course, this argument does not imply that feature-based algorithms should be 
used: it says, however, that locations of edges (meant here as sharp changes 
in intensity) contain most of the correct information. 

3.1. Computing the Minimal Optical Flow 

Consider a rigid surface S moving in space from (2.2.1). The image irradiance 
E at the time t at the point (x p ,y p ) is equal to the scene radiance L at the 
point (a, 6) on S, i.e. E(x p ,y p ,t) = L(a,b). The image irradiance at the 
time t + At is given by the scene radiance of the surface at the time t + At. 
As shown in Figure 3, the point 1 on S that radiates toward {x p ,y p ) at the 
time t + At is the point (a - Aa,b - Ab).l The normal N to S at the time 
t + At at the point (a - Aa, b - Ab), ~N t +At{a - Aa, b - Ab), will be 

N t+ At(a - Aa, b - Ab) = N t {a - Aa, b - Ab) + AN (3.1.1) 



lWe assume that the surface corresponds to a moving convex body to avoid self- 
occlusions due to the motion. In fact, the computation that follows holds for 
any convex surface patch. 
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Figure 3 Computing the minimal optical flow: the point (a, b) on 5 radiates toward 
(x p ,y p ) at time t. The point (a — Aa,b — Ab) radiates toward the same point at 
time t + At. The normal Ni is the normal to the S at the point (a,b) and N2 at 
(a - Aa,b - Ab). 



where AN is the first order variation of N due to the motion of S during 
the time interval At. Now in the case of translation 

AN = (3.1.2) 

while in case of rotation with angular velocity u 

AN = ux NA* (3.1.3) 

Notice that (3.1.3) can be considered as the expression of AN for any 
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kind of motion. Similarly, for each argument A of the scene radiance, we can 
write 

A t+At {a,b) = A t (a,b) + AA. (3.1.4) 

To compute A^4, let us distinguish between arguments of L that are 
intrinsic function of the surface coordinates (a, b) , such as texture and albedo, 
and those that are in fact function of the space coordinates (x,y, z), (such 
as the illumination and the point of view) and that are expressed in terms 
of (a, b) only for convenience. If A is an intrinsic function of the surface 
coordinates, it follows immediately that 

AA = 0, (3.1.5) 

while if A is a function of the space coordinates, from the Taylor expansion 
we have 

AA = VAvAt, (3.1.6) 

where V is the gradient operator with respect to the space coordinates. Let 
us assume that L can be written as a function of m arguments A\i = 1, ..., m 
and of N. Then, taking into account (3.1.3) and (3.1.4), (2.2.1) becomes 
E{x p ,y p ,t + At) = L(A\{a - Aa,b - Ab) + AA 1 ,N t {a - Aa,b - Ab) + AN) 

(3.1.7) 
at time t + At and 

E{x p ,y p ,t) = L(A\{a,b),N t {a,b)) (3.1.8) 

at time t. Therefore, using (3.1.6) and (3.1.7), 

HE - 

dt 

lim ^-[L(A\U- Aa,b- A6) + AA\N t {a- Aa,b - A6) + AN)- 
At — At \ v 

-L(A\(a,b),N t {a,b))), (3.1.9) 

where the AA 1 are computed using (3.1.5) or (3.1.6) according to the kind 
of argument. From (3.1.9), the minimal optical flow can be derived easily. 
To simplify notation, let us suppress the subscript t from Equation (3.1.9). 
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From (3.1.9) we easily get 

dE „ (da db\ Aai„ Ai Jr dL 

s \ Ai' Ai I /-»■ f)A. RN 



dt s Kdt'dt) *-" dAi dN 

1=1 

if p of the A l (i — l,...,m) require the use of (3.1.6) to compute VA and 

Therefore, using (2.1.6), (2.1.7), and (2.2.4), we can write 

v± ~ Of = JT ^ VA l ■ v + ~ ■ to x N (3.1.10) 

Thus, the normal components of the two fields are different if the surface 
undergoes a motion with a rotational component, or the reflectance function 
contains arguments depending on space coordinates. 

Let us consider now some interesting examples in detail. 



3.2. Translation of a Lambertian Surface 

Consider a lambertian surface S. The scene radiance due to S will be 

L = pl-N (3.2.1) 

where p is the albedo of 5, I the unit vector in the direction of the illumina- 
tion and N is the unit normal to the surface. Let us compute the difference 
(3.1.10) between the normal components of the optical flow and of the mo- 
tion field corresponding to a translation of S in space with velocity v under 
uniform fixed illumination. Substituting (3.2.1) in (3.1.10) and changing the 
sign, we have 

v± = O f , (3.2.2) 

since u> = and none of the arguments of L in (3.2.1) depends on space con- 
straints (I is constant). Therefore, the minimal optical flow of a translating 
lambertian surface uniformly illuminated is exactly equal to the motion field. 
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Remark: in the case of non-uniform illumination the right hand side of (3.2.2) 

contains an extra term due to AI. Using (3.1.6) to compute the components 

of AI, (3.1.10) yields 

1 idldx dl dy dl dz \ 

V ^-° F = f^E\\ p \d-xtt + Yy"dt + d~zH)' N ' 
which can be rewritten 

||Vpjc/|| at 
since dl/dt = (the illumination is supposed to be fixed). Let us consider 

now the case of a rotating lambertian surface. 

3.3. Rotation of a Lambertian Surface 

Let S be a lambertian surface rotating in space with angular velocity w. Let 
I be again uniform. Applying the same argument of the previous section but 
taking into account the constraint (3.1.3) for V7V, we get 

pN • I x u , . 

v± -Op = ,,- F|| • (3.3.1) 

|| Vpii|[ 

In the case of rotation, therefore, even under uniform illumination, the 
minimal optical flow and the normal component of the motion field are dif- 
ferent. They are equal for any surface only if u and I are parallel. This 
corresponds to the case of a surface rotating around an axis parallel to the 
direction of uniform illumination. In the case of non-uniform illumination, 
an extra term like the one in (3.2.3) must be added to (3.3.1). Remark: it 
is worth considering analytically the example of the rotating sphere of the 
previous section. Due to rotational symmetry we have 

N(o- Ao,6- A6) +cj xN(o- Aa,b- Ab)At = N(a,6) (3.3.2) 

Va,V6 on the sphere. Furthermore, 

It+At{a ~ Aa, b - Aft) = h(a - Aa, b - Ab) + AI = I t (a, 6), (3.3.3) 
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since in this case the displacement in space, vA£, is equal to the displacement 
on the surface, (Aa, Aft). Therefore, if p is uniform, 

-f=0. (3,3,) 

The minimal optical flow is then, as expected, equal to zero under any illu- 
mination. 

3.4. Translation of a Specular Surface 

Let us consider now a model of reflectance more realistic than the lambertian 
one. Following Phong (1975; see also Horn and Sjoberg, 1979) we define the 
scene radiance as a linear combination of a lambertian and a specular term, 
i.e. 

L = L lamb + L spec- (3-4.1) 

The lambertian term is equal to the one used before, while the specular term 

E=^. (3.4.2) 

where s is the fraction of light reflected by the surface, D = /n + x is the 
vector pointing from the focus to the radiating point and 

R = I-2(I-N)N (3.4.3) 

is the unit vector in the direction of the perfect specular reflection. Let us 
assume that 5 is not a function of the direction of the incident light and that 
it is constant on the surface. The specular term is thus proportional to the 
cosine of the angle between the direction of specular reflection and the line 
of sight. 

Since we are computing derivatives and L is a linear combination of 
Liamb and Lgp e c we can compute separately the contributions to the min- 
imal optical flow due to the lambertian and the specular term, adding the 
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results afterward. Therefore, we only need to compute now the specular one. 
Let us consider, first, the case of pure translation of a surface S radiating 
accordingly to (3.4.2) and let us call 5 a specular surface. If S is translating 
with velocity v and I is uniform, substituting (3.4.2) into (3.1.10) and taking 
into account the constraint (3.1.2), we have 



s (D 2 vR-(D-v)(D-R)) 
vi-Of^-^ jj^jj , (3-4-4) 

since from (3.1.6) 



AD dD dx dD dy dD dz dD dx ln A . 

lim = (- H = = — = v. (3.4.5) 

At— o At dx dt dy dt dz dt dt dt 

Using again the two fields we get a well known vector identity: 

.,-0, = £ (TX ?»'£i XD) . (3-4.6) 

D A \\\ P E\\ 

Thus, in the case of translation of a specular surface, the minimal optical 
flow and the normal component of the motion field are always different. 

Remark: let us consider the case of orthographic projection. When / — > oo, 
(3.4.6) becomes 

v± - O f = 0, 

since when / — ► oo, D — * oo. Therefore, in the othographic limit, the 
minimal optical flow of a translating specular surface is equal to the normal 
component of the 2-D motion field. 

3.5. Rotation of a Specular Surface 

Consider now the same specular surface S rotating in space with angular 
velocity u. Then, substituting (3.4.2) into (3.1.10) and taking into account 
the constraint (3.1.3), we have 
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v_l-O f = ^(2Z? 2 ((I-N)(D-wxN) + (D-N)(I-w x N))- 

||V p E||D 3 V v 

-((w xx) x D) -(R x D) + 

((wxx ) xD)-(RxD)), (3.5.1) 

since v = u x x and Xo gives the location of the axis of rotation. Now 
(3.5.1) gives 

v± - F = ^—~ (2D 2 (I ■ N)(D • w x N) + 2D 2 (D ■ N)(I • u x N) + 

||V P E||D 3 V v n ' 

+L> 2 ((u> xx)-R) + (D-R)(D-w xx) + 2£> 2 ((u; x x ) xD) • (R x D)) 
This expression can be simplified in the following way: since D = /n+x, 

vi-O f = ^-^ (2fD 2 {I ■ N)(n •wxN) + 2D 2 (rN)(x'WX N) + 

+2£> 2 (D-N)(I-u;) + 2 J D 2 ((a; x x ) x D) 
(RxD)), 

that can be rearranged to give 

v± - F = ^-^ (f(n x w) ■ (2£> 2 (I • N)N + (D • R)x) + 

||V p E||D 3 v v ' \ y > v ) j 

+J) 2 (Ixw) -(2(D-N)N-x) +2£> 2 ((w x x ) xD) -(RxD)), 
but x = /n — D; therefore, 



v ± -0 F = ^— ^ f(n x w) • (2/L» 2 (I • N)N + /(D • R)D) + 

||V p E||D 3 V v i \ j k i j i 

+ (I x w) • (2£> 2 (D • N)N - D 2 D + fD 2 n) + 2£> 2 ((w x x ) xD)-(RxD) 
That is, 
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v ± - F = S —^ f(n x w) • (2/£> 2 (I ■ N)N + /(D • R)D) + 

||V p Ei|D 3 V v ; v ' 

+ (I x w) • ( - £> 2 (D - 2(D • N)N) + /£> 2 n) + 2D 2 ((w x x ) x D) • (R x D)) . 

Since 

(Ixw) -N= -(N x w)-I, 

we have 



v ± -0 F 



|V P E 



—3 ((n x u) ■ {2fD 2 {I ■ N)N + /(D • R)D - fD 2 l) + 



+ (I x w) • ( - £> 2 (D - 2(D • N)N)) + 
+2£> 2 ((w x x ) x D) • (R x D)) , 

but I - 2(1 • N)N = R, and so 



v±-Of= n3 ,iy7 m ^ X w ) • ( D " 2 (° ■ N ) N )" 
1/ [| \ phi l J \ 

-/(n x w) • (D x (D x R)) + 2D 2 ((u; x x ) x D) • (R x D)). (3.5.2) 

The minimal optical flow, therefore, is equal to the motion field for any 
specular surface only when I, u) and n are parallel. 

Remark: let us consider, again, the orthographic limit. Taking into account 
that as / — > 00, D — > 00 and D/D — > n, (3.5.2) becomes 

v ± -O f = , )r7 2j L„ ((n-N)(a; x I • N) + (I-N)(w xn-N)). (3.5.3) 

\\vpE\\ 

Therefore, even under othographic limit, the two fields are different. 



3.6. General Case 

Let us consider, now, the general case. We will assume (3.4.1) as scene 
radiance of a surface S undergoing a given rigid motion (composition of a 
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rotation and a translation) in space. Adding together (3.3.1), (3.4.6) and 
(3.5.2), we obtain the difference between the motion field and the minimal 
optical flow for a surface in the general case under uniform illumination, i.e.: 



pN • I x u s (v x D) • (R x D) 

V± — Op = — h 



\V P E\\ D 3 ||V P £|| 

+ z?3||v p £i| ( D2(I X w) " (D " 2( ° ' N)N) ~ 

-/(n x w) • (D x (D x R)) + 2L> 2 ((w x x ) x D) • (R x D)) . (3.6.1) 

The right-hand side of (3.6.1) is generally different from zero. In fact, 
there are no general conditions under which it is identically equal to zero. 
Notice, however, that if ui and v are bounded 

lim \v±-O F \ = 0, (3.6.2) 

|V,,E| — oo 

Equation 3.6.2 shows that the points in the image where the gradient is 
stronger are the points where the minimal optical flow is closer to the motion 
field. These points are characterized by sharp changes in intensity - edges 
-, that usually correspond to important physical events on surfaces, such 
as boundaries, orientation discontinuities and especially surface markings. 
Thus, to solve problems such as structure from motion, or the recovery of 
the 3-D velocity field, which require an accurate estimate of the 2-D motion 
field, edge-based algorithms seem more suitable than algorithms based on 
spatial and temporal derivatives of the image brightness. As a consequence, 
in order to obtain a precise reconstruction of the 2-D motion field, algorithms 
based on the solution of the correspondence problem among edges may be 
used. Notice that matching can be best performed between frames that are 
closely spaced in time whereas the structure from motion computation is 
best performed between widely spaced frames. The whole argument agrees 
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with the fact that, as intuitively expected, the minimal optical flow and the 
motion field at image features corresponding to precise locations on the 3-D 
surfaces coincide. It must be pointed out that in this analysis we have not 
considered shadows and self-shadow effects. They usually give rise to edges 
in the image that do not correspond to features in the scene. Furthermore, 
the Phong model of reflectance does not include sharp intensity changes due 
to specularities. 



4. Qualitative Properties of the Minimal optical flow 

Traditionally, the optical flow has been considered as the first step for recov- 
ering 3-D structure and 3-D motion. In this chapter we suggest a different 
use of the minimal optical flow. We argue that qualitative properties of the 
2-D motion field give useful information about the 3-D velocity and the 3-D 
structure of surfaces and that these qualitative properties can be usefully 
inferred from the obtainable minimal optical flow. As an example of this 
approach, we introduce the qualitative properties associated with 2-D dy- 
namical systems and show how to process minimal optical flow and motion 
field for making them equivalent to flows of dynamical systems on the plane. 
We then suggest, from properties of structural stability of dynamical sys- 
tems, that the minimal optical flow may be equivalent to the motion field in 
terms of qualitative properties. 

4.1. What is the minimal optical flow for? 

In the previous section we have shown that the minimal optical flow and 
the motion field are different almost everywhere. As a consequence, the 
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minimal optical flow cannot be used to solve problems such as structure 
from motion and recovery of the 3-D velocity field, whose solutions rely on 
precise reconstruction of the 2-D motion field. We have also proved that the 
two fields are very similar at locations where the image gradient is strong. 
This led to the suggestion that feature-based algorithms may provide more 
reliable solutions to those problems. 

Here we argue that the minimal optical flow, as a field defined almost 
everywhere, can be used to retrieve meaningful information about the 3-D 
velocity field and the 3-D structure of the scene. In particular, we consider 
qualitative properties of the 2-D motion fields which can be connected to 
significative events in the scene. Such properties are likely to be found in 
the corresponding minimal optical flows as well. As an example, consider an 
object moving toward the image plane. This kind of motion generates a focus 
of expansion in the 2-D motion field. The presence of a focus of expansion 
on the image plane, therefore, may be related to an object moving toward 
the plane itself. As we have seen, however, the information available is not 
the motion field, nor its normal component, but the minimal optical flow (or 
its nomal component). If the difference between the two fields is sufficiently 
small, we expect to find a focus of expansion also in the minimal optical flow. 
In the next sections we will show how the 2-D motion field and the optical flow 
can be considered vector fields tangent to flows of some dynamical system: it 
becomes then possible to establish a suggestive analogy between the theory 
of structural stability of dynamical systems and the qualitative description 
of the two fields. A focus of expansion of a dynamical system, for example, 
is a stable property for small perturbations of the system: this means that 
given a vector field with a focus of expansion, every field obtained from it by 
means of a sufficiently small perturbation will also show a focus of expansion. 
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4.2. Smoothing the Optical Flow and the Motion Field 

In order to establish a connection with the theory of stability of dynamical 
systems, we must insure that the optical flow and the motion field have an 
appropriate degree of smoothness. This is not always the case, because of 
discontinuities arising at object boundaries or to noise affecting the optical 
flow data. We suggest to use a filtering step to smooth the field. It is 
worthwhile noticing that a filtering step on the normal component of a dense 
motion field is a (regularization) method to recover the whole 2-D motion 
field. 2 

4.3. Qualitative Descriptions of Dynamical Systems 

For a rigorous and thorough review on dynamical systems see Hirsch and 
Smale (1974). Here, for the sake of completeness, we summarize the main 
definitions and results. 

A dynamical system is a C 1 map <fr: R x A —> A, where A is an open 
set of an Euclidean space and writing 4>(t,x) = 4>t{x), the map <j>t'. A — > A 
satisfies: 

(a) 4>q: A — * A is the identity; 

(b) the composition <j>t(<t>s{x)) — 4>t+s for each t,s G R. 

A dynamical system <f>t on A gives rise to a differential equation on A, 



that is a vector field y: A —> E defined as follows: 

dt 



y(x) = ^-Jt(x) . (4.3.1) 

t=o 



This "smoothed" 2-D motion field may not be the same recovered using standard 
algorithms, but its qualitative properties are likely to be preserved. The analogy 
we are about to present, indeed, will support this argument (and the equivalence 
between qualitative properties of the 2-D motion field and the optical flow as 
well). 
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Thus, for every x, y(x) is the tangent vector to the curve t — ► 4>t(x) at t = 0. 
Equation (4.3.1) can be rewritten in a more conventional way as 

f = V(x). (4.3.2) 

Under suitable conditions on y(x), there exists a dynamical system associ- 
ated to (4.3.2) as a differential equation. Namely, a sufficient condition on 
y{x) is that it is a C 1 function defined on an open subset of R 2 . Intuitively a 
dynamical system can be thought as a one-parameter family of transforma- 
tion 4>t\ A — >• A describing the motion of the points in A as the time passes. 
The trajectories of the points are given by the solution curves to equation 
(4.3.2). Since equation (4.3.2) is autonomous (that is, the right-hand side 
does not depend explicitly on time), if y(x°) = 0, then x = x° is a solution 
to it. Without loss of generality, we can assume that x° coincides with the 
origin. For obvious reasons, we will restrict our attention to planar systems, 
(i.e. in what follows, A will be an open set in R 2 ). Solutions like x° are 
called equilibrium points or equilibria. In the case of linear systems, useful 
qualitative information about the behaviour of the solution to (4.2.2) can be 
obtained from the eigenvalues of the matrix M of the coefficients of the differ- 
ential equation. The restriction to planar systems reduces the classification 
to four fundamental cases: 

I : M has real eigenvalues of opposite signs. In this case the origin is called 
a saddle: the equilibrium is unstable (an equilibrium is stable if any 
nearby solutions to it stays nearby for all the future time. It is unstable 
otherwise). 

// : The eigenvalues have negative real parts. The origin is called a sink and 
it is stable equilibrium. The main property of a sink is that 

lim x(t) = 

t— >oo 
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Figure 4. Vector field tangent to a planar sink: all the solutions curves are pointing 
toward the origin. 

Qualitatively, the phase portrait of the solutions, that is, the family of the 
solutions curves as a subset of R 2 , looks like Figure 4, where only some 
tangent vectors of some solutions curves have been drawn. Sinks can be 
classified depending on further characteristics of the eigenvalues. A focus 
(Figure 4), for example, represents the case of coincident eigenvalues 
(M is supposed to be diagonalizable); a node, the case of different real 
eigenvalue; a spiral, the case of complex conjugates eigenvalues. A sink- 
increasing rotational component corresponds to each different case. 

Ill : The eigenvalues have positive real parts. The origin is called a source. 
The main property of a source is that 



and 



lim \x(t)\ = oc 

t—>oo 



lim |x(OI =0. 

t— ► — oo 



A source can be considered as the dual case of a sink: the phase portrait 
of a source and of the correponding sink are the same except that for the 
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direction of the motion which must be reversed. Reversing the arrows 
in Figure 4, for example, obtain the phase portrait of a system with 
coincident real positive eigenvalues. A source is obviously an unstable 
equilibrium. 

IV : The eigenvalues are pure imaginary. The origin is called a center. All 
the solutions are periodic with the same period. A center is a stable 
equilibrium. For a reason that will be made clear soon, this last case is 
of little practical interest, since even a small perturbation of the field will 
make the orbits spiral inward to (or outward from) the origin, changing 
the qualitative properties of the solution's curves. In other words, a 
center is not a structurally stable property. 

The crucial point is that this classification is exhaustive. Every solution 
to Equation (4.3.2) (in the linear case) looks like a saddle, a sink, a source, or 
a center. The same classification holds for the non-linear case with respect to 
the eigenvalues of the derivative of the right-hand side of (4.3.2), considered 
as a linear operator. This is equivalent to consider a linear approximation 
of the system in the neighbors of the origin. However non-linear systems 
are interesting in theirself, since they can show also a different qualitative 
behavior. A non-linear system, indeed, can have in addition limit cycles. 
Intuitively, a limit cycle is a closed orbit towards which other solutions' 
curves spiral with the same asymptotic period. Defining a u> -limit set, L ul (x), 
as L UJ (x) = {a <E A such that 3t n — > oo with x(t n ) — > a} and similarly an 
a-limit set L a {x) as L Ql (x) = {b £ A such that 3t n — > — oo with x(t n ) — ► b}, 
a limit cycle is a closed orbit 7 such that 7 C L L0 (x) or 7 C L a (x) for some 
x = 7. Under somewhat more restrictive conditions, a limit cycle can be a 
periodic attractor (for a rigorous definition of it see Hirsch and Smale 1974). 
Intuitively, a periodic attractor is a limit cycle such that nearby trajectories 
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not only have the same asymptotic period but also are in phase. Saddles, 
sinks, sources and periodic attractors are very important for a qualitative 
description of planar systems. Indeed it can be shown that such properties 
are structurally stable, that is they persist after a perturbation of the right- 
hand side of (4.3.2). As far as planar systems are concerned they also fully 
characterize limit sets. By means of the Poincare-Bendixon theorem it can 
be shown that compact limit sets other than limit cycles are saddles, or sinks, 
or sources or trajectories joining them. 

4.4. Equilibria and their Interpretations 

In the definition of dynamical system the right-hand side of Equation (4.3.2) 
can be interpretated as a vector field tangent to the family of curves in the 
plane, solutions to (4.3.2) itself. It is straightforward to see that both the 
smoothed optical flow and motion field (i.e. after the filtering operation) can 
be considered as istances of such a vector field 3 . Indeed, it is sufficient to 
insure that both the fields are continuous with continuous first derivatives. 
The classification of the solutions can now be interpreted in terms of char- 
acteristic points of the 2-D motion field. A source, for example, corresponds 
to a focus of expansion of the field. The structural stability of the source, 
in turn, says that a focus of expansion persists even if the field is perturbed. 
From this perspective a focus of expansion is expected to be detectable in 
a 2-D motion field reconstructed with different algorithms and in the opti- 
cal flow as well, when they can be considered as perturbed examples of the 



3 We stress the fact that the analogy with the dynamical system is between phase 
portraits of dynamical systems and motion flows. The parameter t in the defini- 
tion of dynamical system is not the physical time. We considered motion flows, 
such as the 2-D motion field or the optical flow at a fixed time, comparing them 
with the vector field tangent to the phase portrait of some system: we are not 
interested in the physical meaning of the underlying dynamical system. 
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"true" 2-D motion field. 

4.5. Discussion 

If our point of view is correct, the only critical property of the optical flow is 
that it have the same qualitative properties of the 2-D velocity field. Notice 
that this requirement also satisfies two important uses of the optical flow: 
to detect discontinuities and to help long-range matching of the stereo type, 
needed for the computation of structure-from-motion. Quantitative equiva- 
lence, which is impossible in general, is in any case irrelevant for this use of 
the optical flow. As a consequence, many different "optical flows" may be 
defined. Equation (2.1.6) does not have any priviliged role: other definitions 
could be preferred on the basis of criteria such as computability (from image 
data) or ease of implementation (for given hardware constraints). 

This point of view has clear implications for biological visual systems: 
movement detecting cells (say, in the retina) do not have to compute the 
specific minimal optical flow defined by equation (2.1.6): other, possibly 
simpler, estimates of the velocity field that preserve its qualitative proper- 
ties are equally good candidates (such as correlation-like algorithms). This 
argument may explain why the models proposed to explain motion dependent 
behaviour in insects (Hassenstein and Reichardt, 1956), motion perception 
in humans (Van Santen and Sperling, 1984) and physiology of cells (Barlow 
and Levick, 1965; Torre and Poggio, 1978) are all implementing computa- 
tions quite different from the minimal optical flow as it is usually defined (see 
equation 2.1.6). In addition all these models do not typically measure ve- 
locity - not even in the case of uniform translation in a frontoparallel plane. 
Even for simple motions of the latter type the output of models such as the 
correlation models depends on both the velocity and the spatial structure of 
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the moving pattern. One is tempted to consider this as a weakness of these 
models compared to the definition of minimal optical flow, Equation 2.1.7. 
Our results, however, show that this is not the case: first, the minimal optical 
flow is correct only in a very special situation; second, all these models may 
have the same qualitative properties of the motion field, which, from our 
point of view, is the only critical requirement for a "good" measurement of 
motion. The next question is of course whether these biological models are in 
fact "close" enough to the motion field to share the same qualitative proper- 
ties. We do not know the answer yet. We conjecture, however, that they are 
indeed usually similar enough to preserve the main qualitative properties of 
the motion field. The conjecture is based on results (Poggio and Reichardt, 
1973 and Poggio, 1985) showing that most of the biological models proposed 
so far can be considered as special instances or approximations of a general 
class of nonlinear models (characterized as Volterra systems of the second 
order); and that the minimal optical flow, as defined in equation, is also 
approximately a Volterra functional of the second order (Poggio, 1985). 

It is important to stress that the approach outlined in the second part 
of this paper for classifying the qualitative properties of the optical flow is 
only one of the possible methods. While we plan to develop further that 
particular approach, others should be explored as well: in particular flows 
that do not correspond to dynamical systems on the plane may be better 
suited for capturing important and stable properties of the velocity field 
such as motion discontinuities. In this case, the classification of qualitative 
properties should take place without a preliminary smoothing operation. 

In addition to the classification of stable qualitative properties of the 
velocity field, much work needs to be done at the level of their interpretation 
in terms of 3-D structure and 3-D velocity. Some of the qualitative properties 
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of the (smoothed) velocity field have an easy interpretation in those terms: 
an obvious example is again a focus of expansion that is usually related 
to "crashing" motion. It is likely that many, more subtle relations exist 
between the qualitative properties of the flow and the underlying 3-D motion 
and structure. For example, preliminary results by Torre et al. (personal 
communication) suggest that the number of focuses in the (smoothed) field 
may be characteristic for the rigidity of motion in the visible scene. 

Finally, we should mention an obvious extension of the approach de- 
scribed in the second part of the paper. We have only considered so far the 
velocity field "frozen" at a given instant of time. The succession of image 
frames provides in fact a time-dependent field: the evolution in time of the 
qualitative properties we have described - how they are created, disappear 
and transform - should be characterized in qualitative terms, for instance 
using the language of catastrophe and bifurcation theory. The use of time- 
dependent fields should be practically much more robust, because of the 
redundant information available in a sequence of very closely spaced frames 
(in time). Our analysis should be extended to qualitative properties that are 
structurally stable not only at a given time but also in the time dependent 
field. 
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4.6. Appendix Al: Perspective and Orthographic Projections 

In this section we explain in more detail the geometry of perspective pro- 
jection used in the paper. Let n be the unit normal to the projection plane 
and / the focal length. In order to obtain the orthographic projection as 
the limit of the perspective one for / — > oo, the focus cannot be located at 
the origin of the system of coordinates. To simplify the geometry without 
losing in generality, let the origin lie on the projection plane. The vector 
pointing from the focus to a point x = (x, y, z) is now /n + x. To obtain the 
expression of the projected point x p notice that from Figure 1 is easy to see 

that 

/n + x _ /n + x p 



From that, we have 



(/n + x) • n / 

, /n + x 



and finally 



or 



/ + x n 

f 

x p = (x - (x • n)n) 

? / + x-n v v ; ; 

f 

X,, = (n x (x x n)). 

p / + x-n v v n 

The orthographic projection equation can be easily obtained for / — > oo, i.e. 

f 

x or t = lim x p = (n x (x x n)) = (n x (x x n)). 

Combining the last two equations, we obtain the general relationship 

between perspective and orthographic projection, that is 

/ 
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